Add the pure query core for the analysis catalog by alex-rawlings-yyc · Pull Request #211 · sillsdev/interlinearizer-extension

alex-rawlings-yyc · 2026-08-10T21:55:44Z

Closes #192.

The pure derivation layer for the Analysis Catalog (#186) — no React, no store, no PAPI, no UI.

Modules

src/utils/search-fold.ts — foldForSearch (NFKD → strip \p{Mn} → NFC → lowercase → fold final letter forms), kept deliberately apart from normalizeSurfaceForm. NFKD folds compatibility distinctions generally — ligatures, width and superscript variants, non-breaking spaces — not just positional shapes.
src/utils/analysis-query.ts — buildCatalogRows, applyCatalogQuery, deriveFacets.
src/store/analysisSlice.ts — selectCatalogRows(state, currentBook), memoized on the two token-level arrays plus the analysis language and book.

Three divergences from the issue, agreed before implementing

Split build from query. Analysis catalog: pure query core (analysis-query.ts, search fold, memoized selector) #192 specifies buildCatalogRows(analysis, query) but also requires search/sort/filter to run outside the memoized build — those conflict. Shipped as buildCatalogRows(analysis, scope) + applyCatalogQuery(rows, query). buildCatalogRows takes Pick<TextAnalysis, 'tokenAnalyses' | 'tokenAnalysisLinks'> so the selector memoizes on those arrays rather than the whole TextAnalysis.
CatalogRow carries pos / features / confidence. The row shape in Analysis catalog: pure query core (analysis-query.ts, search fold, memoized selector) #192 has none of them, so deriveFacets(rows) could not derive the facets it is asked to. status and lexicon refs are deliberately not facets: status is constant by construction (only approved links count as usages) and lexicon-ref presence is a boolean, not a value list.
No analysisLanguages in the query. searchText folds every gloss value present, so a gloss in a language the project no longer declares stays findable.

#192 and #186's row model should be amended before #193 starts, since #193 consumes these exports.

Not in the issue, but the data requires it

USJ verse SIDs are verbatim, so a bridged verse arrives as "GEN 1:3-4:0" and Number('3-4') is NaN. Usage parsing goes through the existing firstVerseNumber; there is a test for it.

Testing

Covers every behavior #192 lists, plus the script-specific cases the search fold turns on. Full suite passing, 100% coverage, lint clean.

🤖 Generated with Claude Code

This change is

Summary by CodeRabbit

New Features
- Added catalog search across glosses and morphemes.
- Added filtering by book, usage, gloss, and morphology.
- Added sorting by usage, text, gloss, and first occurrence.
- Added catalog facets for books, part of speech, confidence, and feature values.
- Search supports case- and diacritic-insensitive matching across Greek, Hebrew, Latin, Devanagari, and Arabic text.
Tests
- Added comprehensive coverage for catalog queries, filtering, sorting, facets, multilingual search, and catalog updates.

coderabbitai · 2026-08-10T21:56:14Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR adds a pure analysis catalog query core, search folding, row construction, facets, filters, sorting, and a memoized Redux selector. It also adds tests for catalog behavior, normalization, query operations, and selector memoization.

Changes

Analysis catalog

Layer / File(s)	Summary
Catalog contracts and row construction `src/utils/analysis-query.ts`, `src/utils/search-fold.ts`, `src/__tests__/utils/analysis-query.test.ts`, `src/__tests__/utils/search-fold.test.ts`, `src/__tests__/utils/analysis-identity.test.ts`	Adds catalog contracts, approved usage grouping, document ordering, row construction, folded search text, and Unicode search normalization.
Facets, filters, and sorting `src/utils/analysis-query.ts`, `src/__tests__/utils/analysis-query.test.ts`	Adds facet derivation, search and filter evaluation, and count-, text-, gloss-, and first-use sorting.
Memoized store selector `src/store/analysisSlice.ts`, `src/__tests__/store/analysisSlice.test.ts`	Exports `selectCatalogRows` and verifies stable references for unchanged state and recomputation after gloss updates.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AnalysisState
  participant selectCatalogRows
  participant buildCatalogRows
  AnalysisState->>selectCatalogRows: token analyses, links, language, current book
  selectCatalogRows->>buildCatalogRows: catalog scope
  buildCatalogRows-->>selectCatalogRows: catalog rows
  selectCatalogRows-->>AnalysisState: memoized row collection

Possibly related issues

Issue 186: The PR implements the Analysis Catalog v1 objectives, including rows, search folding, facets, filters, sorting, and memoized selection.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	Most [`#192`] requirements are covered, but the summary specifies compatibility decomposition instead of the required NFD normalization for foldForSearch.	Implement foldForSearch with NFD normalization, combining-mark removal, NFC recomposition, and lowercasing while preserving the required search-only distinctions.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Out of Scope Changes check	✅ Passed	All listed changes support [`#192`], including implementation, selector integration, and focused tests; no unrelated code changes are identified.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: adding the pure query core for the analysis catalog.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch analysis-catalog-derivation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

alex-rawlings-yyc

@alex-rawlings-yyc resolved 2 discussions.
Reviewable status: 0 of 7 files reviewed, all discussions resolved (waiting on alex-rawlings-yyc).

imnasnainaec

Synthesized review, not human-checked.

Authored by Claude Opus 5 (1M context) (claude-opus-5[1m]) running in Claude Code. It is a synthesis of Devin's review of this PR: every finding was re-read against the branch, and the ones that were stale, incorrect, or purely informational were dropped — but no human has verified what remains. Treat each comment as a suggestion to evaluate rather than a request.

Behavioral claims were checked by running the actual fold pipeline and Canon.bookIdToNumber on Node 22; the measured values are quoted inline. Comments prefaced with :pick: are low-priority or optional.

Row derivation, search fold, and facets, memoized as selectCatalogRows. Row building and query application are separate functions so searching and sorting never rebuild rows; the search fold stays apart from normalizeSurfaceForm, since folding a diacritic into identity would silently merge distinct analyses.

Adds pos, confidence, and feature filters, treats an empty selection as inactive, and orders the confidence facet by strength. The search query is trimmed in applyCatalogQuery, not in foldForSearch, whose output separates the row fields a match must not span.

Compatibility normalization covers the shapes Unicode encodes as variants; Greek and Hebrew final letters, which it encodes as distinct letters, are mapped.

Ties fell through to payload creation order, so a draft of mostly one-usage rows was ordered by when each analysis happened to be recorded; they now fall back to form, then gloss.

The analyses and links these fixtures build gained required createdAt / updatedAt, so each now spreads FIXTURE_STAMPS. Also drops two fixture counts from the doc comments, which nothing kept in sync.

Absence now lists as undefined after a field's values, so a field one analysis is tagged with still raises a dropdown, and a search for what an analysis lacks is possible. Also folds a pasted Windows line ending in a catalog search.

Usage counts match the suggestion pool's frequency only while a token carries at most one approved link, and a single-book draft is offered no books facet even when unused rows sit alongside the used ones.

Cache the fold per analysis record, count a token once however many approved links carry it, and keep a field's own line break from spelling the separator a match may not span.

alex-rawlings-yyc

Addressed all comments left

@alex-rawlings-yyc made 12 comments.
Reviewable status: 0 of 7 files reviewed, 11 unresolved discussions (waiting on alex-rawlings-yyc and imnasnainaec).

A gloss of only whitespace counts as missing, the reading the rest of the analysis layer takes, and the query is trimmed after folding, by which point a spacing diacritic has decomposed to a bare space.

imnasnainaec

@imnasnainaec reviewed 4 files and all commit messages, made 2 comments, and resolved 8 discussions.
Reviewable status: 4 of 7 files reviewed, 6 unresolved discussions (waiting on alex-rawlings-yyc).

src/utils/search-fold.ts line 11 at r5 (raw file):
I find this comment to be unclear as to whether the mapping is to or away from the word-final form. Perhaps something like:

Letter shapes that are only used at the end of a word, mapped to the default shape used everywhere else.

src/__tests__/utils/analysis-query.test.ts line 190 at r5 (raw file):

  });

  // The morpheme form is suppletive, so a match cannot come from the surface form instead.

⛏️ I don't find this comment helpful.

Only CR and LF collapsed, so a gloss broken with a vertical tab or a line separator was unfindable by a spaced query. Also settles the direction of the final-letter map and narrows the usage-count and fold-cache docs to what they guarantee.

alex-rawlings-yyc

@alex-rawlings-yyc made 6 comments.
Reviewable status: 4 of 7 files reviewed, 6 unresolved discussions (waiting on alex-rawlings-yyc and imnasnainaec).

src/__tests__/utils/analysis-query.test.ts line 190 at r5 (raw file):

Previously, imnasnainaec (D. Ror.) wrote…

⛏️ I don't find this comment helpful.

Deleted.

src/utils/search-fold.ts line 11 at r5 (raw file):

Previously, imnasnainaec (D. Ror.) wrote…

I find this comment to be unclear as to whether the mapping is to or away from the word-final form. Perhaps something like:

Letter shapes that are only used at the end of a word, mapped to the default shape used everywhere else.

Fair — the old subject was the letter, which has both shapes, so it left the key
ambiguous. Making the word-final shapes the subject settles it.

imnasnainaec

@imnasnainaec reviewed 3 files and all commit messages, and resolved 6 discussions.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on alex-rawlings-yyc).

alex-rawlings-yyc self-assigned this Aug 10, 2026

alex-rawlings-yyc marked this pull request as draft August 10, 2026 21:56

This comment was marked as outdated.

Sign in to view

alex-rawlings-yyc commented Aug 11, 2026

View reviewed changes

alex-rawlings-yyc mentioned this pull request Aug 11, 2026

Analysis catalog: pure query core (analysis-query.ts, search fold, memoized selector) #192

Closed

alex-rawlings-yyc force-pushed the analysis-catalog-derivation branch from 35e2980 to 7e0162a Compare August 11, 2026 18:44

alex-rawlings-yyc marked this pull request as ready for review August 11, 2026 18:45

This comment was marked as outdated.

Sign in to view

imnasnainaec reviewed Aug 12, 2026

View reviewed changes

alex-rawlings-yyc added 15 commits August 12, 2026 08:21

Let a search match a letter in any positional shape

f15a96b

Compatibility normalization covers the shapes Unicode encodes as variants; Greek and Hebrew final letters, which it encodes as distinct letters, are mapped.

Document the catalog row cache by what callers can rely on

0d0f501

Keep a search from matching across two catalog fields

c1d4d79

Hold the catalog listing still where a sort key ties

3e5b915

Ties fell through to payload creation order, so a draft of mostly one-usage rows was ordered by when each analysis happened to be recorded; they now fall back to form, then gloss.

Stamp the catalog fixtures so they typecheck again

de1730b

The analyses and links these fixtures build gained required createdAt / updatedAt, so each now spreads FIXTURE_STAMPS. Also drops two fixture counts from the doc comments, which nothing kept in sync.

Make an untagged analysis a facet choice of its own

e051402

Absence now lists as undefined after a field's values, so a field one analysis is tagged with still raises a dropdown, and a search for what an analysis lacks is possible. Also folds a pasted Windows line ending in a catalog search.

Name the conditions the catalog docs hold under

e55559d

Usage counts match the suggestion pool's frequency only while a token carries at most one approved link, and a single-book draft is offered no books facet even when unused rows sit alongside the used ones.

Sort an analysis with no gloss to the end of the listing

220cd09

Document what a catalog row's feature map holds

abbdd3a

Make a new catalog sort key fail to compile until it is ordered

d8fc717

Document that the search fold keeps an eszett distinct

9c97eff

Catch a search fold that drops a letter it should keep

9288c6d

Link the identity fold the search fold warns against

753e4fa

alex-rawlings-yyc force-pushed the analysis-catalog-derivation branch from 7187fd9 to 753e4fa Compare August 12, 2026 14:21

Fold each catalog analysis's search text once

4096003

Cache the fold per analysis record, count a token once however many approved links carry it, and keep a field's own line break from spelling the separator a match may not span.

alex-rawlings-yyc commented Aug 12, 2026

View reviewed changes

Take whitespace at the catalog's edges as no text at all

94ff883

A gloss of only whitespace counts as missing, the reading the rest of the analysis layer takes, and the query is trimmed after folding, by which point a spacing diacritic has decomposed to a bare space.

imnasnainaec reviewed Aug 13, 2026

View reviewed changes

Comment thread src/__tests__/utils/analysis-query.test.ts Outdated

imnasnainaec reviewed Aug 13, 2026

View reviewed changes

alex-rawlings-yyc commented Aug 13, 2026

View reviewed changes

Comment thread src/__tests__/utils/analysis-query.test.ts Outdated

Comment thread src/utils/analysis-query.ts

Comment thread src/utils/analysis-query.ts

Comment thread src/utils/analysis-query.ts Outdated

imnasnainaec approved these changes Aug 13, 2026

View reviewed changes

alex-rawlings-yyc enabled auto-merge (squash) August 13, 2026 19:35

alex-rawlings-yyc merged commit 8708024 into main Aug 13, 2026
13 of 14 checks passed

alex-rawlings-yyc deleted the analysis-catalog-derivation branch August 13, 2026 19:36

alex-rawlings-yyc mentioned this pull request Aug 14, 2026

Analysis catalog: search, sort, filter, and row windowing #231

Open

coderabbitai Bot mentioned this pull request Aug 14, 2026

Add the read-only analysis catalog panel #234

Open

Uh oh!

Uh oh!

Conversation

alex-rawlings-yyc commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Modules

Three divergences from the issue, agreed before implementing

Not in the issue, but the data requires it

Testing

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

alex-rawlings-yyc left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

imnasnainaec left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-rawlings-yyc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

imnasnainaec left a comment

Choose a reason for hiding this comment

Uh oh!

alex-rawlings-yyc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

imnasnainaec left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

alex-rawlings-yyc commented Aug 10, 2026 •

edited

Loading

coderabbitai Bot commented Aug 10, 2026 •

edited

Loading